feat: rai_bench #436
Merged
feat: rai_bench #436
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
```
Exception in thread Thread-1 (spin):
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/executors.py", line 294, in spin
self.spin_once()
File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/executors.py", line 794, in spin_once
self._spin_once_impl(timeout_sec)
File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/executors.py", line 786, in _spin_once_impl
self._executor.submit(handler)
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
```
Co-authored-by: Jakub Matejczyk <jakub.matejczyk@robotec.ai> chore: naming changes Co-authored-by: Jakub Matejczyk <jakub.matejczyk@robotec.ai> chore: naming changes Co-authored-by: Jakub Matejczyk <jakub.matejczyk@robotec.ai> chore: redefine benchmark model Co-authored-by: Jakub Matejczyk <jakub.matejczyk@robotec.ai> Co-authored-by: Kacper Dąbrowski <kacper.dabrowski@robotec.ai> refactor: SceneConfig to BaseModel class feat: add SceneSetup to store initial scene setup build: poetry initialization of rai_benchmarks and rai_simulations chore: add licence lines build: create packages from rai_benchmarks and rai_simulations chore: removed mistakenly added file feat: scene config implementation Add O3DEEngineConnector features Signed-off-by: Kacper Dąbrowski <kacper.dabrowski@robotec.ai> Remove an unused file Signed-off-by: Kacper Dąbrowski <kacper.dabrowski@robotec.ai> Add binary path caching Signed-off-by: Kacper Dąbrowski <kacper.dabrowski@robotec.ai> Add two example scenes Signed-off-by: Kacper Dąbrowski <kacper.dabrowski@robotec.ai> feat: replace binary path with ros2 launch command + binary path refactor: renamed rai_sim and rai_bench fix: fixed shutdown of binary chore: make pose mandatory chore: remove rai_bench because it is developed on another branch ci: add missing license lines laoding and spawning frm benchmark rabsed naming change grab xyz benchmark benchmarks run new sim every scenario for now
removed name in launch_robotic_manipulation to avoid misnaming nodes
add main to run bechmark add helper functions to Task
task takes simulation config as arg now adjusted typing
adjusted paths to configs add move arm request to O3DEngineArmManipulationBridge adjusted main to task naming change
task defined more clearly
This fixes bug when get_transform returns always the same, first transform Add logs to track retrieved positions
added log when all scenarios finished
refactored scenario to inlucde config path so it can be logged easily
maciejmajek
pushed a commit
that referenced
this pull request
Feb 26, 2025
Co-authored-by: Maciej Majek <maciej.majek@robotec.ai> Co-authored-by: Bartłomiej Boczek <bartlomiej.boczek@robotec.ai> Co-authored-by: MagdalenaKotynia <magdalena.kotynia@robotec.ai> chore: pre-commit
maciejmajek
pushed a commit
that referenced
this pull request
Mar 7, 2025
Co-authored-by: Maciej Majek <maciej.majek@robotec.ai> Co-authored-by: Bartłomiej Boczek <bartlomiej.boczek@robotec.ai> Co-authored-by: MagdalenaKotynia <magdalena.kotynia@robotec.ai> chore: pre-commit
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Link to the previous PR which contains review comments: #426
Purpose
Add Test Benchmark for O3DE simulation
Add framework for easier benchmark creation
Proposed Changes
Majority of file changes are only formatting changes, main changes are in src/rai_bench package and src/rai_sim
This PR contains new package rai_bench. It introduces classes like Task, Scenario and Benchmark(src/rai_bench/rai_bench/benchmark_model.py), which are frame for creating benchmarks and it also contains O3DE Test Benchmark which is a test implementation of a benchmark for O3DE engine simulation.
O3DE Test Benchmark (
src/rai_bench/rai_bench/o3de_test_bench/), contains 2 Tasks(tasks/) - GrabCarrotTask and PlaceCubesTask (these tasks implement calculating scores) and 4 scenes(configs/) for O3DE robotic arm simulation.Example of how can scenarios and benchmark be defined and run can be found in
rai_bench/rai_bench/main.pyAdditional changes:
Issues
Testing
the binary to simulation can be downloaded from here: humble
To run this benchmark you need o3de bridge config file in
src/rai_bench/rai_bench/o3de_test_bench/configs/o3de_config.yaml, this config file should include fields:to install and run:
Logs about the progess of benchmark can be found in 'src/rai_bench/o3de_test_bench/benchmark_agent.log'
You should be able to see:
After finishing all scenarios, simulation should close and all nodes and services should also shutdown after no more than 30 seconds